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FOREWORD 


This  report  was  prepared  as  a  joint  effort  of  the  Behavioral  Sciences  Laboratory,  6570th 
Aerospace  Medical  Research  Laboratories]  and  the  Educational  Science  Division  of  U  S. 
Industries,  Inc.  working  under  the  terms  of  Contract  No.  AF  33(616)-6983.  The  data  on  which 
the  report  is  based  were  gathered  during  the  period  January  to  June  1961  at  Keesler  Air  Force 
Base,  Mississippi  with  the  participation  and  support  of  the  3380th  Technical  School,  Air  Training 
Command.  The  authors  are  Dr.  Felix  F.  Kopstein,  who  also  served  as  Air  Force  contract 
technical  monitor  during  this  time  period,  Dr.  Virginia  Zachert,  Project  Field  Director  for 
Educational  Science  Division,  and  Major  Richard  T.  Cave,  Operator  Training  Section,  Behavioral 
Sciences  Laboratory,  the  current  contract  technical  monitor.  Mr.  James  D.  Gilmour,  Educational 
Science  Division,  contributed  greatly  by  providing  the  Analysis  of  Variance  computation  and 
assisting  in  the  editing  and  revision  of  the  report. 

The  work  was  performed  under  Task  171101,  "Development  and  Evaluation  of  a  Prototype 
Automated  Technical  Training  Course"  of  Project  1711,  "Automated  Training  Materials. "  This 
report  is  in  response  to  Headquarters  USAF  Operational  Support  Directive  No.  362,  dated  11  June 
1959,  "Automated  Training  Materials.  " 
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Mr.  A.  Overmeyer,  Chief  Instructor,  Department  of  Communications  Electronics 
Principles,  Keesler  AFB,  Mississippi 

Mr.  J.  Stewart,  Instructor,  Department  of  Communications  Electronics  Principles, 

Keesler  AFB,  Mississippi 

S/Sgt  R.  Steinmetz,  Instructor,  Department  of  Communications  Electronics  Principles, 
Keesler  AFB,  Mississippi 

Lt  Esco,  Project  Liaison  Officer  (ATC),  Keesler  AFB,  Mississippi 

Lt  E-  Benson,  Research  Assistant,  Behavioral  Sciences  Laboratory,  6570th  Aerospace 
Medical  Laboratory  (on  detached  duty  at  Keesler  AFB,  Mississippi) 

Mr.  N.  Crowder,  Principal  Investigator  and  Technical  Director,  Educational  Science 
Division,  U-S-  Industries,  Inc.,  New  York,  New  York 

Mr.  P.  Pipe,  Program  Editor,  Educational  Science  Division,  U  S.  Industries,  Inc., 
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ABSTRACT 


This  field  study,  conducted  at  Keesler  Air  Force  Base,  Mississippi,  constituted  a  prelimi¬ 
nary  evaluation  of  intrinsic  programming  for  automated  training.  Automated  instructional  materi¬ 
als  used  during  the  first  6  weeks  of  the  Communications  Electronics  Principles  course  were 
presented  to  beginning  electronics  students  via  35-mm  film  on  the  AutoTutor  Mark  I,  a  rearview 
projection  machine.  Using  the  Keesler  Mathematics  Test,  three  groups — Experimental,  Control, 
and  Blind  Control — were  selected  and  matched  from  the  middle  ability  range  of  each  of  two 
entering  classes.  The  Experimental  group  received  via  the  machines  all  instruction  normally 
received  through  lecture  and  discussion.  However,  they  followed  the  usual  method  for  their 
practical  problems.  The  students  using  machines  learned  adequately  from  this  experimental 
program.  The  interpretations  of  these  results  and  implications  for  Air  Force  training  are 
discussed. 


PUBLICATION  REVIEW 
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PRELIMINARY  EVALUATION  OF  A  PROTOTYPE 
AUTOMATED  TECHNICAL  TRAINING  COURSE 


INTRODUCTION 


The  Air  Force  interest  in  automated  training  can  be  traced  back  at  least  to  1951.  Beginning 
then  and  until  the  middle  1950's,  the  Maintenance  Laboratory,  Air  Force  Personnel  and  Training 
Research  Center,  Lackland  Air  Force  Base,  Texas,  devoted  considerable  resources  in  developing 
automated  devices  and  techniques  to  improve  the  training  of  Air  Force  maintenance  personnel.  A 
number  of  prototype  automated  trainers,  preliminary  programs,  and  proposed  experiments  were 
developed  by  representatives  of  this  organization  (refs.  1,  2,  5,  6-10,  14).  After  this  research 
group  was  deactivated  in  1958,  many  of  these  research  people  continued  their  work  with  automated 
instruction  and  programmed  learning  in  widely  scattered  groups.  *  Because  of  increasing  interest 
in  the  possibilities  of  auto-instructional  techniques,  a  conference,  called  Automatic  Teaching  of 
Verbal  and  Symbolic  Skills,  was  held  at  the  University  of  Pennsylvania  in  1958,  under  the  sponsor¬ 
ship  of  the  Air  Force  Office  of  Scientific  Research  (ref.  12).  At  this  conference,  the  discussions 
by  research  workers  who  had  been  actively  experimenting  with  automated  training  techniques 
strongly  suggested  the  feasibility  of  their  application  on  a  large  scale.  Accordingly,  plans  for  a 
large-scale  field  test  that  had  been  proposed  informally  a  short  time  before  were  reconsidered. 

In  June  1959,  Headquarters  USAF  issued  Operational  Support  Directive  No.  362  for  the 
"Development  and  Evaluation  of  a  Prototype  Automated  Technical  Training  Course"  to  the  Air 
Force  Systems  Command  (AFSC),  It  was  assigned  to  the  Training  Research  Branch,  Behavioral 
Sciences  Laboratory,  6570th  Aerospace  Medical  Research  Laboratories  for  execution.  This 
directive  specified  generally  the  development  of  automated  ". .  -  teaching  materials  with  promising 
potential  for  reducing  instructor  requirements  and  training  costs."  Supplementary  instructions 
indicated  the  desire  for  an  objective  evaluation  of  this  potential  as  well  as  comparisons  between 
the  effects  of  automated  and  conventional  training.  Accordingly,  plans  were  prepared  to  field-test 
an  intrinsically  programmed  technical  training  course  to  assess  its  ability  to  develop  proficient 
graduates  as  compared  to  that  of  current  conventional  instruction  in  this  same  course.  Additional 
objectives  concerned  comparative  evaluations  of  costs,  required  training  time,  instructor  require¬ 
ments,  interchangability  of  instruction,  device  requirements,  and  so  forth. 

Initial  instructions  from  Hq  USAF  provided  for  the  joint  conduct  of  the  project  by  the  Air 
Training  Command  (ATC)  and  the  Air  Force  Systems  Command.  Hie  Air  Training  Command  was 
assigned  the  responsibility  for  providing  a  test  site  (school)  and  vehicle  (course),  for  furnishing 
students  and  instructors,  and  for  furnishing  general  logistic  support  to  personnel  collecting  evalua¬ 
tion  data.  The  Air  Force  Systems  Command  was  charged  with  arranging  the  preparation  of  the 
automated  materials,  providing  all  technical  monitorship  during  their  development,  furnishing  the 
technical  direction  for  the  subsequent  evaluation,  and  for  drafting  of  reports. 


*Of  the  several  excellent  bibliographies  available  which  list  these  early  Air  Force  efforts,  the 
reader  is  referred  to:  Goebel,  L.G. ,  Capt,  Air  Force  Activities  in  Programmed  Learning  - 
A  Working  Bibliography,  Hq  Office  of  Aerospace  Research,  Washington  25,  D.C.,  February 
1962. 
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The  development  of  the  automated  course  version  and  the  preparation  of  all  requisite 
materials  were  delegated  to  the  Educational  Science  Division  (ESD)  of  U.S.  Industries,  Inc. , 
under  Contract  AF  33(616) -6983.  Educational  Science  Division  also  provided  a  small  field  staff 
at  the  designated  test  site  to  pretest  newly  prepared  automated  materials  so  that  any  gross  short¬ 
comings  could  be  identified  and  eliminated  in  revisions.  Further,  the  field  staff  was  to  be  respon¬ 
sible  for  suggesting  evaluation  plans  and  for  collecting  the  evaluation  data. 

The  choice  of  a  suitable  test  site  and  a  suitable  test  vehicle  was  influenced  by  the  necessity 
of  conducting  the  entire  field  test  without  seriously  upsetting  established  routines  in  Air  Training 
Command  technical  schools  and  without  affecting  the  required  rate  of  production  of  graduates  in  a 
given  specialty.  These  considerations  coupled  with  the  need  for  a  constant  availability  of  appre¬ 
ciable  numbers  of  students  led  to  the  selection  of  the  Communications  Electronics  Principles 
course  (ABR  30020)  taught  at  the  3380th  Technical  School,  Keesler  Technical  Training  Center, 
Keesler  Air  Force  Base,  Mississippi.  The  Communications  Electronics  Principles  (CEP)  course 
is  the  basic  technical  course  for  all  of  the  specialties  in  Air  Force  electronics.  Hence,  it  assured 
a  substantial  student  flow  over  the  period  during  which  the  evaluation  would  be  carried  out.  The 
other  determining  factor  was  the  willingness  of  school  authorities  to  cooperate  in  the  project. 

The  completion  of  conventional  instruction  in  course  ABR  300  20  normally  requires  19  weeks. 
Academic  instruction  is  divided  into  ten  instructional  blocks  totaling  570  hours.  The  course 
organization  is  as  follows: 

Instructional  Blocks  Subjects  Duration  in  Hours 


Block  I 

Direct  Current 

60 

Block  n 

Alternating  Current 

60 

Block  in 

Reactive  Circuits 

60 

Block  IV 

Principles  of  Vacuum  Tubes  and  Transistors 

60 

Block  V 

Special  Purpose  Tubes 

30 

Block  VI 

Amplifiers  and  Oscillators 

90 

Block  VH 

Special  Circuits 

90 

Block  VIII 

Motors  and  Servo  Mechanisms 

30 

Block  IX 

Multivibrators  and  Sweep  Circuits 

60 

Block  X 

Microwave  Principles 

30 

All  specialties  in  the  electronic  career  field  complete  the  first  six  blocks  of  instruction.  After 
the  sixth  block  has  been  completed,  certain  students  destined  for  one  particular  specialty  leave 
the  course  (see  figure  1)  for  specialized  training.  Other  specialties  drop  out  in  a  similar  fashion 
after  the  seventh  and  eighth  blocks.  The  remainder  of  the  students  complete  all  ten  blocks  of 
instruction - 

The  cooperative  nature  of  the  project  and  the  geographical  dispersion  of  the  major  partici¬ 
pants  required  that  a  formal  organization  with  clearly  defined  functional  responsibilities  be  set  up 
Headquarters  Air  Training  Command  at  Randolph  Air  Force  Base,  Texas  handled  all  administra¬ 
tive  arrangements  for  the  actual  conduct  of  the  field-test.  The  Behavioral  Sciences  Laboratory 
assumed  the  technical  management  and  direction  as  well  as  administration  of  the  contract  with  , 
the  Educational  Science  Division.  Educational  Science  Division  of  U  S.  Industries,  Inc.,  then  at 
Goleta,  California  assumed  responsibility  for  the  writing,  editing,  and  production  of  the  auto¬ 
mated  course  materials  and  for  the  delivery  of  this  material  to  Keesler  Air  Force  Base.  The 
Project  Office  at  Keesler  was  jointly  staffed  by  personnel  of  Air  Training  Command,  Air  Force 
Systems  Command,  and  Educational  Science  Division. 
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Figure  1.  Training  Schedule  Basic  Technical  Training 
(Airman  Electronics)  ABR  30020 
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APPROACH  AND  METHOD  OF  EVALUATION 


Rigney  and  Fry  (ref.  17)  have  recently  surveyed  and  catalogued  existing  auto-instructional 
materials  and  programming  formats.  Kopstein  and  Shillestad  (ref.  15)  have  discussed  the 
varieties  of  programming  formats  and  their  significance  in  auto-instruction.  It  will  be  clear 
that  only  one  type  of  programming  format  could  be  evaluated  in  a  single  field-test  unless  a  truly 
gigantic  effort  were  to  have  been  initiated.  The  format  chosen  for  this  purpose  was  Crowder's 
"intrinsic  programming"  (ref  4). 

The  Intrinsic  Programming  Technique 

Norman  A.  Crowder,  originator  of  this  technique,  conceived  of  it  when  faced  with  the  problem 
of  training  airmen  to  efficiently  diagnose  and  correct  malfunctions  in  electronic  circuitry.  This  is 
fundamentally  a  process  of  sequential  decision-making  in  which  a  single,  most  efficient  sequence 
can  be  designated.  Although  any  disclosure  resulting  from  a  departure  from  the  most  efficient 
sequence  does  not  provide  much  information  in  the  formal  sense  and  is  considered  largely  redun¬ 
dant,  it  is  also  obviously  helpful  to  a  student.  This  observation  contains  the  germ  of  Crowder’s 
programming  format,  which  is  characterized  by  the  two-way  responsiveness  typically  found  in 
the  pupil-tutor  relationship. 

Intrinsic  programming  is  an  appropriate  title  for  this  technique  since  the  exact  program 
sequence  is  determined  by  factors  that  are  intrinsic  to  the  particular  student  and  the  particular 
learning  situation.  The  original  strategy  or  mechanism  for  intrinsic  programming  is  the  scrambled 
book.  On  page  one  of  this  book  (or  on  the  first  frame  of  microfilm,  if  the  material  is  photographi¬ 
cally  recorded)  the  student  is  introduced  to  the  topic  with  a  short  informational  paragraph.  This 
is  followed  by  a  key  question  and  a  choice  of  several  possible  answers.  Each  answer  choice  is 
indexed  to  a  page  number,  or  to  an  image  number  if  microfilm  is  used.  While  the  sequence  of 
page  numbers  in  the  scrambled  book  is  in  normal  order,  there  is  no  topical  continuity  from  page 
to  page.  Only  when  the  student  turns  to  the  page  number  corresponding  to  his  answer  choice,  will 
he  find  there  a  meaningfully  related  statement.  Assuming  that  his  choice  is  incorrect,  he  will 
find  on  that  page  (or  image)  in  the  following  order:  (a)  a  recapitulation  of  his  answer  choice, 

(b)  a  statement  that  his  answer  is  incorrect,  (c)  a  brief  paragraph  reviewing  the  presumably 
faulty  reasoning  that  led  to  this  answer  choice  together  with  corrective  information,  and 
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(d)  directions  to  return  to  the  first  page.  If  it  can  be  assumed  that  on  his  second  attempt  the 
student  makes  a  correct  choice  of  answers,  he  will  find  on  the  corresponding  page  the  main 
continuation  of  the  topical  material-  There  will  be  (a)  a  recapitulation  of  the  answer  choice, 

(b)  a  statement  that  his  answer  is  correct,  (c)  a  review  of  the  reasoning  that  leads  to  the 
correct  answer,  (d)  an  extrapolation  of  this  reasoning,  which  introduces  new  information, 

(e)  a  key  question  about  the  new  material,  and  (f)  a  choice  of  several  possible  answers.  For 
more  detailed  discussions  of  the  intrinsic  programming  format,  see  Galanter  (ref.  12)  and 
Lumsdaine  and  Glaser  (ref.  16). 

When  the  instructional  material  is  recorded  on  microfilm,  it  is  projected  for  actual  use 
on  the  ground-glass  screen  of  a  device  known  commercially  as  the  Mark  I  AutoTutor.  This 
device  also  permits  rapid  selection  of  any  frame  desired  for  viewing.  In  this  study,  the  pro¬ 
grammed  material  was  recorded  on  35-mm  microfilm  and  presented  to  the  students  in  the  Mark  I 
AutoTutor.  The  material  was  prepared  and  filmed  in  15  copies  and  these  films  were  sent  to 
Keesler  Air  Force  Base  where  they  were  inserted  into  the  Mark  I  as  needed.  One  of  the  advan¬ 
tages  of  this  device  is  that  it  produces  a  permanent  tape  (made  by  a  printer  in  the  machine)  that 
lists  the  number  of  each  frame  chosen  by  the  student  and  the  amount  of  time  (in  tenths  of  minutes) 
elapsing  between  each  choice.  Thus,  a  complete  record  of  the  presumably  significant  aspects  of 
each  student's  learning  behavior  is  recorded  and  can  be  subjected  to  statistical  analysis. 

Development  of  the  Automated  Course 

When  the  Communications  Electronics  Principles  course  was  originally  chosen  for  the  field 
study,  we  did  not  know  that  it  was  being  revised  as  of  1  January  1960.  This  proved  to  have  its 
advantages  and  its  disadvantages.  To  obtain  the  first  materials  on  the  course,  it  was  necessary 
for  the  Research  Engineer  to  attend  classes  for  the  first  five  blocks  to  ascertain  their  precise 
content,  since  the  printed  materials  for  the  course  were  largely  unavailable.  As  there  could  be 
no  deviation  from  the  content  covered  in  the  course  at  Keesler,  it  was  necessary  to  analyze  it 
carefully  with  respect  to  detailed  topics,  terminology,  sequence  of  concepts,  relationships  among 
concepts,  and  so  forth,  and  decide  how  to  best  present  this  instruction  information  using  the 
intrinsic  programming  technique. 

The  initial  programming  proved  to  be  much  more  difficult  and  time  consuming  than  had  been 
anticipated.  The  Keesler  schools  do  not  teach  the  course  content  of  the  first  4  weeks  with  the 
same  terms  and  techniques  as  do  most  colleges-  This  proved  confusing  to  the  technical  writers 
who  constituted  the  programming  staff.  They  were  shortly  replaced  by  professional  journalistic 
writers  who  knew  little  or  no  electronics,  but  at  least  did  not  have  to  unlearn  antagonistic  pre¬ 
conceptions. 

The  evaluation  of  the  programmed  material  was  begun  in  July  1960  when  the  first  draft  of 
block  I  course  material  (on  microfilm)  was  used  for  training  students  at  Keesler.  The  data 
collected  for  each  student  from  the  AutoTutor  tapes  were  tallied  daily  by  individual  items.  When 
the  students  had  completed  each  block,  these  tallies  were  forwarded  to  the  editorial  staff  of 
Educational  Science  Division  to  guide  the  revision  of  the  material. 

During  the  fall  of  1960,  the  first  two  blocks  of  material  were  revised  and  new  copy  was 
made  available  for  the  field  study,  which  began  in  January  1961.  The  first  draft  of  block  III  was 
added.  Unfortunately,  problems  in  the  filming  of  the  revised  materials  resulted  in  mediocre 
optical  presentation  by  means  of  the  Mark  I  AutoTutor. 

Routine  Teaching  of  the  Communications  Electronics  Principles  Course 

All  enlisted  men  entering  the  30XXX  career  field  of  the  Air  Force  must  first  pass 
through  the  CEP  course.  Most  of  the  men  enter  directly  from  Lackland  AFB,  Texas,  where 
they  have  completed  part  of  their  basic  military  training  (the  last  6  weeks  are  completed  after 
entering  technical  school  at  Keesler).  Another  large  group  enters  from  other  career  fields  or 
Air  Force  specialties,  often  those  which  have  become  obsolete.  All  of  these  men  must  have  a 
minimum  score  of  70  on  the  Electronic  Aptitude  Index.  Past  studies  by  the  Personnel  Laboratory  at 
Lackland  (before  the  time  when  enlistees  were  tested  at  recruiting  stations)  have  shown  that  the 
Electronic  Index  has  a  validity  of  between  .  60  and  .  70  for  the  various  30XXX  schools  at  Keesler 
(ref.  3). 
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Approximately  200  students  enter  the  CEP  course  each  week.  These  are  divided  into  three 
slutts:  from  0600  to  1200  hours  for  A-shift;  from  1200  to  1800  hours  for  B-shift;  and  1800  to  2400 
hours  for  C-shift.  In  each  of  these  shifts  there  are  up  to  five  sections  of  not  over  20  students  per 
section.  The  classes  begin  on  Wednesday  of  each  week.  To  group  the  students  homogeneously 
in  the  various  sections  of  each  shift,  the  CEP  administrative  personnel  developed  the  Keesler 
Math  Test.  This  is  a  50-item  test  that  was  developed  to  identify  students  who  are  extremely  weak 
in  math  as  well  as  those  who  are  extremely  strong.  The  top  20  percent  (as  of  June  1961)  are 
placed  in  an  accelerated  section  that  covers  the  first  three  blocks  of  material  (regularly  6  weeks) 
in  4  weeks,  while  the  low  20  percent  are  placed  in  an  improvement  section  that  covers  the  same 
amount  of  material  in  8  weeks.  The  other  three  sections  are  scheduled  for  the  regular  6  weeks 
(figure  2). 


REGULAR  COURSE 


10  Days 
Direct  Current 

10  Days 

Alternating  Current 

10  Days 

Reactive  Circuits 

IMPROVEMENT  TRAINING 

15  Days 
Direct  Current 

12  Days 

Alternating  Current 

13  Days 

Reactive  Circuits 

Study  Habits  Reading  Improvement 

Remedial  Math.  Special  Assignments 


6  Weeks 


8  Weeks 
(Bottom 
1/5  of 
Class) 


ACCELERATED  PROGRAM 


6  Days 

7-1/2  Days 

6-1/2  Days 

Direct  Current 

Alternating  Current 

Reactive  Circuits 

Less  Drill 

More  Rapid  Presentation 

Increased  Demonstrations 

4  Weeks 
(Top  1/5 
of  Class) 


Figure  2.  Three  Speed  Levels  in  Branch  I  (Blocks  I-m) 
Branch  I  -  Fundamentals 


It  is  from  the  three  middle  sections  that  the  automated  training  section  and  its  matched 
control  groups  were  chosen.  The  A-shift  time  period  was  used  for  the  automated  study,  so  that 
the  A-shift  study  hall  from  1730  to  1900  hours  might  also  be  used. 

In  the  regular  training,  each  student  is  issued  a  student  handbook  and  a  student  workbook. 
The  handbook  is  his  textbook  and  the  workbook  contains  problems  to  be  worked  out  each  day .  If 
students  fail  to  keep  up  with  their  workbook  or  make  low  grades  on  either  daily  quizzes  or  block 
tests,  they  are  assigned  by  the  instructor  to  supervised  study  hall. 
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In  regular  training  for  the  first  three  blocks  of  material,  approximately  20  percent  of  the 
teaching  time  and  testing  grades  are  made  up  of  practical  work.  These  are  experiments  or 
practical  problems  that  the  student  performs  as  outlined  in  his  workbook.  Each  room  is  equipped 
with  the  necessary  equipment  for  the  experiments  constituting  part  of  the  work  of  that  week.  As 
the  students  progress  from  block  to  block  of  instruction,  they  move  from  one  room  to  another  in 
order  to  have  the  proper  equipment.  Every  room  is  adequately  furnished  so  that  each  student  does 
his  own  work  with  his  own  equipment. 

At  the  end  of  each  block,  the  students  are  tested  by  the  Instruction  and  Measurement  Branch. 
The  practical  tests  are  given  on  one  day  (usually  on  Fridays)  and  on  Tuesdays  all  the  written  tests 
are  given.  The  written  exams  are  developed  in  several  series  and  usually  consist  of  50  items 
appearing  on  spearate  5-  by  8-inch  sheets  of  paper  bound  in  booklets.  Each  booklet  begins  with 
a  different  item.  The  items  are  reevaluated  periodically  and  new  tests  are  developed  from  over 
10,000  item-card  files  maintained  by  this  section.  The  passing  score  on  these  tests  is  set  at  78. 

This  figure  usually  eliminates  the  bottom  6  to  7  percent.  The  Evaluation  Division  of  the  school 
does  an  exceptional  job  of  test  development  and  test  examination  in  keeping  the  items  and  tests 
valid  and  reliable.  Maintaining  normal  curves  by  adjusting  item  difficulty  is  one  of  this  Division's 
most  important  tasks. 

In  considering  a  student's  grade,  it  is  important  to  remember  that  it  represents  the  final 
school  grade  and  not  the  individual  block  grades.  If  at  any  time  a  student  is  washed-back  (fails 
a  block),  his  grade  for  that  block  is  the  one  he  receives  when  he  finally  graduates  from  it.  In 
effect,  the  final  course  grade  is  the  grade  that  a  student  obtains  when  he  has  been  allowed  the 
necessary  number  of  wash-backs,  or  additional  weeks  of  training,  to  finish  the  course  and  not 
the  one  he  would  have  received  from  a  course  of  normal  length.  A  normal  attrition  or  wash-back 
rate  in  the  CEP  course  is  approximately  one  or  two  students  per  section  per  block.  This  is 
especially  true  for  the  first  five  blocks. 

The  content  of  the  automated  training  was  nearly  identical  with  that  of  the  conventional  train¬ 
ing.  The  major  difference  was  that  the  students  used  the  machines  instead  of  having  lectures  and 
demonstrations.  An  instructor  was  assigned  to  the  automated  class.  He  was  available  at  all  times 
for  individual  help  and  at  times,  when  there  were  either  errors  in  the  Tutor  Film  or  omissions  of 
material,  he  would  discuss  this  with  the  entire  class.  Each  day,  the  homework  assignments  for 
the  course  were  made  and  the  workbooks  checked.  Students  getting  behind  on  the  AutoTutor, 
failing  to  do  their  homework,  or  making  low  grades  were  assigned  to  study  hall.  This  was  con¬ 
ducted  each  evening  by  the  project  engineer  in  the  machine  classroom  and  not  in  regular  study  hall. 

Students  receiving  automated  training  were  selected  from  the  middle  achievement  group  on 
the  Keesler  Math  Test.  They  were  assigned  the  regular  practical  problems,  did  the  regular 
homework  assignments,  and  were  given  the  regular  block  examinations.  As  part  of  this  field 
study,  the  experimenters  attempted  to  fit  into  the  regular  administrative  schedule  and  frame-work 
of  the  school  but,  at  the  same  time,  to  secure  the  advantages  of  the  self-pacing  aspects  of  automated 
training.  The  wide  range  of  materials  covered  in  a  day  or  a  week  by  the  different  students  was  a 
major  problem.  This  meant  that  during  the  6  weeks  of  this  study,  it  was  not  unusual  to  have  students 
working  on  instruction  that  belonged  in  at  least  3  different  weeks  and  involved  different  practical 
problems  and  required  different  classrooms  having  the  appropriate  equipment.  At  times,  this 
meant  that  the  regular  instructor  required  two  assistants  to  help  the  students  with  their  practical- 
work  and  that  additional  rooms  or  equipment  had  to  be  obtained  for  use.  It  also  meant  that,  on 
occasion,  testing  could  not  be  accomplished  on  the  most  timely  basis. 

Evaluation  Procedure 


To  evaluate  the  automated  training  materials,  machines,  and  techniques,  two  pilot  studies 
were  completed  in  the  summer  and  fall  of  I960.  Gilmore  et  al.  (ref.  13)  have  reported  on 
some  of  this  work.  These  studies  are  summarized  but  briefly  here.  The  materials  were  in  draft 
form  and  a  complete  item  analysis  was  made  of  every  choice.  This  was  used  in  rewriting  the 
material,  which  was  made  available  at  the  beginning  of  1961.  The  AutoTutor  Mark  I  machines 
proved  themselves  rugged  and  capable  instruments.  Moreover,  the  project  engineer  had  learned 
the  machine  problems  unique  to  this  climate,  i.e. ,  salt  in  the  air  and  its  effect  on  the  machines. 
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The  administrative  and  instructive  procedures  had  been  modified  to  better  utilize  the  automated 
training  potential  within  the  total  school  organization.  It  is  probable  that  by  January  of  1961, 
when  the  first  study  (Study  A)  began,  the  automated  training  section  was  accepted  by  the  students 
and  school  administration  as  routine  and  not  appreciably  different  from  the  kind  of  deviation 
introduced  by  the  accelerated  classes  (top  20  percent)  and  the  improvement  classes  (lower  20 
percent)  of  the  A -shift. 

The  great  interest  created  by  the  project  generally  continued,  not  only  on  the  Base  but  in 
the  Air  Force.  However,  the  operation  had  now  become  a  routine  matter.  During  the  spring  of 
1961,  there  were  teams  of  visiting  dignitaries  from  other  branches  of  the  Armed  Forces,  from 
other  countries,  and  from  the  Congress.  Numerous  training  directors  of  large  industries, 
especially  those  assigned  responsibility  for  the  development  of  training  programs  under  various 
government  contracts,  came  to  observe  automated  training  and  to  estimate  its  potential  value. 
Students  were  so  used  to  observers  that  even  a  USAF  General  Officer  did  not  cause  a  stir. 

The  objective  of  this  study  was  primarily  to  obtain  a  "power"  comparison  of  automated  and 
conventional  training.  The  assumption  was  made  that  the  CEP  Department,  because  of  its  exten¬ 
sive  background  in  conducting  military  electronics  training  and  because  of  the  long  experience  of 
each  member  of  its  teaching  staff,  operated  at  the  peak  of  effectiveness  attainable  for  conventional 
training.  How  would  automated  training  compare  with  respect  to  its  power  to  produce  graduates 
of  acceptable  proficiency?  Could  a  highly  complex  technical  course  be  taught  without  lectures? 

A  secondary,  but  possibly  even  more  important,  objective  was  to  obtain  data  concerning  the 
characteristics  of  automated  training  per  se.  Might  not  its  novelty  and  the  publicity  surrounding 
it  induce  a  Hawthorne  effect?*  Would  students  tire  of  too  much  reading?  What  administrative 
problems  might  be  raised  by  automated  training  and  how  could  one  cope  with  them?  Finally, 
what  further  research  and  evaluation  would  suggest  itself? 

To  find  the  answers  to  these  questions,  the  present  field  study  was  begun  in  January  1961. 

The  automated  training  material  was  then  available  for  the  first  three  blocks  or  6  weeks  of  the 
course.  The  first  class,  consisting  of  14  students  in  each  of  the  Control  sections  and  in  the 
Experimental  section  (Study  A)  was  selected  on  the  basis  of  scores  on  the  Keesler  Math  Test  and 
began  work  on  12  January  1961.  To  increase  the  size  of  the  sample,  a  second  class  (Study  B)  was 
selected  immediately  after  the  first  class  completed  the  three  blocks,  and  on  1  March  1961  this 
second  class  began  its  training. 

Students  in  the  Experimental  (automated)  group  and  in  the  Control  group  (aware  of  participa¬ 
tion  in  an  experiment)  were  selected  by  means  of  the  Keesler  Math  Test-  An  equated  group  was 
chosen  by  the  same  means  from  the  B-shift  (1200  to  1800  hours)  to  serve  as  a  Blind  Control  group 
(i  e. ,  unaware  of  participation  in  an  experiment)  which  was  known  only  to  the  project  personnel. 
Since  the  Keesler  Math  Test  is  not  extremely  sensitive  in  the  mid-range,  a  more  sensitive  measure, 
the  Psychological  Corporation  Electronic  and  Physical  Sciences  Aptitude  Test  (EPSAT),  was 
administered  to  these  and  other  incoming  students  for  the  CEP  course.  This  test,  especially  the 
first  two  sections,  has  proved  to  have  satisfactory  validity  for  the  first  three  blocks  of  material 
and  is  now  being  used  in  other  studies. 

The  selection  of  incoming  students  for  the  present  study  was  restricted  to  personnel  of  the 
Air  Force,  Air  National  Guard,  and  Air  Reserve.  Allied  students  were  excluded  from  considera¬ 
tion.  Non-commissioned  officers  were  also  excluded  from  the  study,  except  for  the  Blind  Control 
group. 

Information  was  also  obtained  on  the  Aptitude  Indices  of  the  students,  although  in  the  pilot 
studies  they  were  not  very  highly  correlated  with  grades  in  the  first  three  blocks  of  the  course. 


♦This  effect  was  first  observed  at  the  Hawthorne  plant  of  the  Western  Electric  Company  in  a 
classic  study  of  methods  for  raising  production.  Any  change  of  work  conditions  seemed  to 
increase  the  productivity  of  the  experimental  group  of  workers.  This  was  interpreted  to 
result  from  the  increased  attention  and  publicity  given  to  these  individuals. 
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A  personal  data  sheet  was  completed  by  each  student  in  the  Control  group  and  in  the  Experi¬ 
mental  group  giving  information  concerning  education  to  date,  home  town,  courses  in  math  and 
physics,  marital  status,  age,  military  rank,  and  experience. 

The  experimental  group  using  the  AutoTutors  automatically  produced  tapes  showing  their 
sequential  choice  of  frames  as  well  as  the  time,  in  tenths  of  minutes,  spent  on  each  frame.  The 
tapes  were  analyzed  each  day  by  the  project  field  staff  and  three  or  four  airmen  awaiting  school 
assignment, who  were  detailed  to  help  tally  choices.  Each  day,  students'  "error-ratios,"  or  total 
number  of  choices  over  number  of  incorrect  choices,  were  determined.  In  the  pilot  studies,  this 
index  had  been  shown  to  give  a  very  reliable  indication  as  to  whether  or  not  students  were  success¬ 
fully  mastering  the  material.  The  correlations  between  error-ratio  and  block  grades  ranged 
between  .  60  and  .80-  An  item  analysis  of  every  choice  made  by  each  student  was  prepared  and 
sent  to  the  editors  in  California  to  be  used  in  further  editing  and  rewriting  the  automated  training 
materials.  The  number  of  observations  obtained  on  each  student  thus  ran  into  the  hundreds  each 
day.  Perhaps  this  is  why  the  error-ratio  score  has  stood  up  in  all  blocks  as  the  one  most  consis¬ 
tent  predictor  of  course  grade. 


RESULTS 


The  first  pilot  research  efforts  have  been  reported  elsewhere  (ref.  13).  These  studies, 
using  only  blocks  I  and  n,  indicated  that  the  basic  groups — Experimental,  Control,  and  Blind 
Control — were  logical  and  feasible  groupings  for  comparison  purposes.  The  Experimental  group 
is  consistently  a  randomly  selected  set  of  14  students  whose  scores  on  the  Keesler  Math  Test  fall 
in  the  middle  60  percent  of  the  distribution.  The  Control  group  is  a  matched  set  (on  the  Keesler 
Math  Test)  of  14  students,  who  were  aware  of  their  participation  in  a  research  project,  but  who 
were  exposed  to  the  regular  course  lecture  techniques.  The  Blind  Control  group  was  a  randomly 
selected  group  of  14  from  the  remaining  middle  60  percent  of  the  distribution  and  were  unaware 
of  their  participation  in  the  study  as  their  names  and  scores  were  known  only  to  the  investigators. 
All  airmen  who  were  selected  for  the  Basic  Electronics  Course  had  Electronic  Aptitude  Indexes 
(Elec  AI)  above  70.  Table  I  indicates  the  mean  Keesler  Math  Scores,  the  mean  Elec  AI  Scores, 
and  the  scores  on  the  Engineering  and  Physical  Science  Aptitude  Test  (EPSAT)  for  the  three 
treatment  groups. 


TABLE  I 

A  COMPARISON  OF  THE  SELECTION  MEAN  SCORES 
FOR  THE  THREE  GROUPS 


Group 

Keesler  Math 

Elec  AI 

EPSAT 

Experimental  -  Study  A 

60.29 

74.64 

61.57 

Control  -  Study  A 

59.00 

81.07 

60.00 

Blind  Control  -  Study  A 

60.14 

81.07 

_ * 

Experimental  -  Study  B 

68.86 

80.00 

59.43 

Control  -  Study  B 

67.00 

75.00 

55.54 

Blind  Control  -  Study  B 

68.86 

80.71 

57.82 

♦EPSAT  was  not  administered  to  this  group. 
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Three  treatment  groups  were  selected  in  January  1961  and  completed  blocks  I,  n,  and  III 
in  their  respective  mode,  hereatter  described  as  study  A.  The  mean  scores  for  these  groups  on 
the  theory  portion  of  the  respective  block  exams  are  indicated  in  table  n. 


TABLE  II 

CRITERIA  SCORES  (THEORY)  FOR  STUDY  A 


GROUP 

BLOCK  I 

BLOCK  II 

BLOCK  III 

MS 

n 

a 

MS 

n 

a 

MS 

n 

a 

Experimental 

82.  57 

14 

4.29 

82.23 

13 

6.26 

83.64 

n 

5.77 

Control 

85.  28 

14 

6.01 

82.38 

13 

6.17 

86.80 

10 

3.36 

Blind  Control 

89.00 

14 

4.08 

83.64 

14 

6.30 

82.92 

12 

6.75 

The  differences  among  means  for  the  three  groups  within  blocks  were  tested  using  the  t-test 
and  appropriate  tables.*  Within  block  I  there  was  a  significant  mean  difference  (at  the  .01  level) 
between  the  Experimental  group  and  the  Blind  Control  group.  All  other  mean  differences  were 
found  to  be  nonsignificant. 

As  indicated  in  table  II,  one  airman  from  the  Experimental  group  and  one  from  the  Control 
groups  washed  back  after  block  I.  After  block  n,  two  more  airmen  were  eliminated  from  the 
Experiment  group,  three  more  from  the  Control  group,  and  two  from  the  Blind  Control  group. 

This  attrition  from  the  beginning  samples  of  14  in  each  group  meant  that  only  11  airmen  com¬ 
pleted  all  three  blocks  (the  complete  study)  in  the  Experimental  group,  10  in  the  Control  group, 
and  12  in  the  Blind  Control  group.  Further,  the  reduction  in  sample  occurred  solely  at  the  bottom 
end  of  the  distribution  of  scores,  for  only  those  failing  the  block  were  eliminated.  For  these 
reasons,  the  mean  scores  were  recomputed  for  the  three  treatment  groups  using  only  those  sub¬ 
jects  who  completed  all  three  blocks  of  instruction  or  the  complete  study.  These  mean  scores 
are  indicated  in  table  m. 


TABLE  m 

CRITERIA  SCORES  (THEORY)  FOR  STUDY  A  -  REVISED 


GROUP 

BLOCK  I 

BLOCK  II 

BLOCK  m 

MS 

n 

a 

MS 

n 

a 

MS 

n 

a 

Experimental 

83.82 

a 

3.76 

83.55 

u 

5.89 

83.64 

ii 

5.77 

Control 

87.70 

10 

4.08 

85.30 

10 

2.91 

86.80 

10 

3.36 

Blind  Control 

89.50 

12 

4. 12 

85.42 

12 

4.60 

82.92 

12 

6.75 

♦See  Johnson,  P  O.,  Statistical  Methods  in  Research,  pp  71-73,  Prentice -Hall,  New  York,  1949. 
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Again,  the  mean  differences  were  examined  by  means  of  the  t-test.  Within  block  I  there  was 
a  significant  difference  (at  the  .05  level)  between  the  Experimental  group  and  the  Control  group, 
and  a  significant  difference  (at  the  .01  level)  between  the  Experimental  group  and  the  Blind  Control 
group.  All  other  mean  differences  were  nonsignificant. 

Study  A  was  replicated  beginning  in  March  1961  using  again  three  treatment  groups  of  size 
14.  The  mean  scores  on  the  theory  exams  for  these  groups  are  indicated  in  table  IV. 


TABLE  IV 

CRITERIA  SCORES  (THEORY)  FOR  STUDY  B 


GROUP 

BLOCK  I 

BLOCK  II 

BLOCK  III 

MS 

n 

a 

MS 

n 

a 

MS 

n 

or 

Experimental 

82.86 

14 

5.29 

80.69 

13 

5.54 

84.67 

9 

6.44 

Control 

87.07 

14 

5.15 

85.57 

14 

3.74 

85.57 

14 

5.45 

Blind  Control 

88.21 

14 

4.10 

86.79 

14 

5.08 

87.21 

14 

6.41 

Again,  mean  differences  were  contrasted  by  using  the  t-test.  A  significant  difference  (at 
the  .05  level)  existed  between  the  Experimental  group  and  the  Control  group  within  block  I.  The 
difference  between  the  Experimental  group  and  the  Blind  Control  group  within  block  I  was  signifi¬ 
cant  at  the  .01  level.  Within  block  II,  significant  differences  existed  between  the  Experimental 
group  and  the  Control  group  (at  the  .05  level),  and  between  the  Experimental  group  and  the  Blind 
Control  group  (at  the  .01  level).  Other  mean  differences  were  nonsignificant. 

The  data  for  study  B  was  recomputed  in  a  similar  fashion  as  for  study  A  to  include  only 
those  subjects  who  completed  all  three  blocks  of  instruction.  These  results  are  shown  in  table  V. 

TABLE  V 


CRITERIA  SCORES  (THEORY)  FOR  STUDY  B  -  REVISED 


GROUP 

BLOCK  I 

BLOCK  II 

BLOCK m 

MS 

n 

a 

MS 

n 

a 

MS 

n 

a 

Experimental 

85.44 

9 

3.61 

83.44 

9 

3.84 

84.67 

9 

6.44 

Control 

87.07 

14 

5. 15 

85.57 

14 

3.74 

85.57 

14 

5.45 

Blind  Control 

88.21 

14 

4. 10 

86.79 

14 

5.08 

87.21 

14 

6.41 

All  mean  differences  were  compared  by  using  the  t-test,  and  all  lacked  significance  at  the 
.05  level. 

Among  the  analyses  performed  principally  for  the  interest  of  the  investigators  was  a  Repeated 
Observations  Analysis  of  Variance,  the  results  of  which  appear  in  table  VI.  It  is  included  here  to 
demonstrate  the  high  F  ratio  obtained  for  within-subject  differences  between  blocks  of  material  and 
in  the  second-order  interaction.  Interactions  are  also  apparent  in  the  graphed  means  presented  in 
figure  3.  Because  the  samples  for  the  two  studies  were  drawn  from  two  different  student  groups 
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(January  1961  and  March  1961  entering  classes),  it  is  recognized  that  the  data  cannot  properly  be 
combined  for  comparison  of  the  treatment  groups  and  that  findings  must  be  interpreted  as  resulting 
from  two  separate,  but  very  similar,  studies. 

TABLE  VI 

ANALYSIS  OF  VARIANCE  FOR  CORRELATED  DATA 


Source  of  Variance 

df 

SS 

MS 

F 

P 

Overall  Between  Subjects 

(83) 

(7408) 

Treatment  Groups 

2 

861 

430.5 

5.35 

<  .01 

Studies 

1 

130 

130 

1.62 

Groups  and  Studies 

2 

143 

71.5 

Error  (S's  w/i  Gps  and  St) 

78 

6274 

80.44 

Overall  Within  Subjects 

(168) 

(1809 

Blocks 

2 

328 

164 

20.22 

<  .01 

Groups  and  Blocks 

4 

64 

16 

1.97 

Studies  and  Blocks 

2 

32 

16 

1.97 

Groups,  Studies,  and  Blocks 

4 

120 

30 

3.70 

<  .01 

Within  Subjects  Error 

156 

1265 

8.11 

Total 

9217 

Ig  =  Experimental  "machine"  Group 
Ic  *  Control  Group 
Ig  *  Blind  Control  Group 


Figure  3.  Means  for  Treatment  Groups  During  Each  Block  for  Studies  A,  B, 
and  Averaged  Over  Studies  A  and  B 
(All  Participating  Subjects) 
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A  block-by-block,  study -by -study  Analysis  of  Variance  yielded  the  F  ratios  presented  in 
table  VII.  It  should  be  noted  that  treatment  group  N's  differ  with  each  analysis,  and  that  in  no 
case  do  significant  F  ratios  reflect  differences  between  Control  and  Blind  Control  treatment  groups. 

When,  as  was  the  case  in  the  earlier  discussion  of  revised  criteria  scores,  individual  means 
are  recomputed  for  each  block  and  each  study  using  only  those  subjects  who  completed  the  three 
blocks  of  material  (ignoring  the  effects  of  selective  wash-back  at  the  lower  end  of  the  distribution), 
the  resulting  means  may  be  plotted  as  shown  in  figure  4. 

TABLE  VII 

ANALYSIS  OF  VARIANCE  BY  BLOCKS  AND  STUDIES  (F  RATIOS) 


REPLICATION 

BLOCK  I 

BLOCK  II 

BLOCK  III 

A 

6. 16* 

.19 

1.43 

B 

4.69* 

5.96** 

.54 

*p  =  <  .05 

**p  -  <  .01 


Study  A  Study  B 


Blocks  Blocks 


Ig  =  Experimental  Group  (Machines) 

Ij,  =  Control  Group 
Ig  =  Blind  Control  Group 

Figure  4.  Means  for  Treatment  Groups  During  Each  Block  for  Studies  A  and  B  -  Adjusted 
(Including  only  those  subjects  completing  material  through  block  III) 


Significant  differences  were  found  that  constantly  favored  the  Control  groups.  However,  the 
revised  data  (tables  III  and  V)  and  the  repeated  evidence  of  interaction  effects  tended  to  show  that 
these  differences  either  were  associated  with  the  introduction  of  the  students  to  automated  instruc¬ 
tion  or  to  differences  caused  by  the  unequal  wash-back  of  subjects  from  the  three  groups.  As  a 
result,  it  is  concluded  that  at  least  by  block  III  relatively  little  difference  in  performance  (as 
indicated  on  the  block  exams)  existed  among  the  three  treatment  groups . 
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The  complete  raw  data  is  tabled  in  the  appendix.  The  error  ratio  scores  for  the  students  in 
the  Experimental  group  continued  to  correlate  significantly  with  examination  grades  (+  .68  and 
+  .89)  thus  upholding  the  finding  of  the  pilot  studies.  Cumulative  time  spent  by  the  student  on  the 
AutoT\itor,  on  the  other  hand,  did  not  show  a  significant  relationship  with  the  criterion  scores. 

A  field  study  always  results  in  unexpected  situations  and  findings.  They  are  often  tragic  if 
one  is  attempting  to  have  controlled  conditions  from  which  statistically  meaningful  data  can  be 
obtained. 

The  wash-bacx  rate  in  the  first  three  blocks  of  the  CEP  school  created  a  problem  in  this 
•  field  study.  Should  the  student  be  continued  on  the  Auto  Tutors,  or  should  he  be  placed  in  a  regular 
class?  It  was  decided  to  place  him  in  a  regular  class  with  no  deficiency  against  his  school  record 
in  order  that  he  might  have  no  recourse  when,  and  if,  he  washed  out  of  the  school.  This  procedure 
continually  reduced  the  sample  size  of  the  Experimental  group,  and  it  may  also  have  affected  the 
performance  of  some  Experimental  students  as  this  school  policy  soon  became  common  knowledge 
among  the  students. 

The  lack  of  a  common  sample  throughout  the  study  caused  trouble  for  the  statistician  and 
also  for  the  administrative  sergeant  who  had  to  program  the  placement  of  students.  Also,  the 
small  sample  of  only  14  in  a  class  caused  much  trouble  when  the  flow  of  students  reached  almost 
100  per  week,  because  the  other  classes  were  required  to  take  an  overload. 

The  comments  by  the  men  who  washed-back  on  the  machines  caused  the  administrative 
officer  to  comment  that  these  men  complained  similarly  to  those  who  had  failed  under  his  good 
instructors.  He  was  careful  to  note  the  difference  between  the  complaints  against  good  instructors 
and  poor  instructors. 

Another  difficulty  )n  the  execution  of  the  field  study  was  that  some  Experimental  subjects 
were  required  to  wait  a  day  or  two  between  their  completion  of  the  instructional  block  and  the 
administration  of  the  test  over  that  block.  The  effect  of  the  intervening  time  may  have  reduced 
their  test  scores,  since  the  auto-instructional  devices  did  not  provide  an  efficient  means  for 
reviewing  the  instructional  material. 

Certain  subjects  of  the  Experimental  groups  were  given  8  hours  of  instruction  on  the  Auto- 
Tutors  per  day  instead  of  the  usual  6  hours.  This  additional  2  hours  was  substituted  for  their 
normal  study-hall  period.  These  students  had  no  difficulty  maintaining  a  steady  rate  of  progress 
over  this  extended  time  period.  Furthermore,  there  were  no  complaints  of  eye  strain  even  though 
the  visual  presentation  of  the  machines  was  marginal. 

Throughout  the  experimental  period,  those  students  whose  placement  grades  were  either  very 
high  or  very  low  were  most  complimentary  in  their  reactions  to  the  AutoTutor  course.  Both  the 
fast  and  slow  students  appeared  to  appreciate  moving  at  their  own  pace.  The  better  students 
commented  about  how  nice  it  was  not  to  have  to  wait  on  those  slow-pokes  and  how  nice  it  was  not  to 
have  to  hear  everything  repeated  a  dozen  times.  The  slower  students  thought  that  it  was  nice  to  be 
able  to  really  understand  something  before  being  hurried  to  the  next  bit  of  new  material.  As  a 
contrast,  the  average  students  tended  to  state  that  the  machines  made  them  work  and  that  they 
preferred  to  have  someone  tell  them  things,  because  that  system  was  easier. 

The  instructor  stated  that  teaching  these  classes  was  much  more  demanding  than  delivering 
a  prepared  lecture,  because  he  had  to  act  as  a  tutor  and  be  available  tohelp  each  student  with  his 
individual  problem.  Other  instructors  not  directly  associated  with  the  AutoTutor  classes  often 
commented  on  the  quietness  of  the  experimental  classroom. 

Students  in  automated  Instruction  may  complete  the  course  of  instruction  in  more  or  less 
time  than  that  allotted  for  conventional  instruction.  In  this  study,  records  were  kept  of  the  time 
to  complete  each  block  of  instruction.  These  data  will  be  presented  in  subsequent  reports.  Such 
data  are  necessary,  of  course,  for  a  complete  evaluation. 
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CONCLUSIONS  AND  IMPLICATIONS 


Considering  all  available  information,  it  seems  justified  to  conclude  that  students  trained 
with  the  AutoTutors  progressed  at  a  rate  and  to  a  level  nearly  comparable  with  the  students 
instructed  in  the  conventional  fashion.  Although  the  examination  scores  for  the  Control  groups 
were  somewhat  higher  than  comparable  scores  for  the  Experimental  AutoTutor  groups,  in  a 
practical  sense,  the  differences  were  not  great.  Further,  statistically  significant  differences 
were  primarily  confined  to  scores  on  examinations  given  after  the  first  instructional  block,  and 
differences  tended  to  be  reduced  by  eliminating  scores  by  subjects  who  washed-back. 

Hie  finding  of  acceptable  progress  by  students  trained  on  the  AutoTutor  appears  to  be 
pertinent  to  a  relatively  wide  range  of  student  ability,  because  the  subjects  used  were  representa¬ 
tive  of  the  middle  60  percent  of  incoming  course  personnel  during  January  and  March  1961.  There 
is  no  reason  to  believe  that  personnel  entering  during  these  periods  were  unusual  or  not  represen¬ 
tative  of  students  taking  the  CEP  course. 

The  results  reported  here  are  based  upon  a  highly  restricted  instructional  situation.  If  this 
kind  of  machine  instruction  were  to  be  used  within  a  school  such  as  that  at  Keesler  on  a  continuing, 
operational  basis,  machines  would  be  available  24  hours  a  day  in  sufficient  numbers  to  satisfy  the 
needs  of  the  training  program.  Further,  an  effort  would  be  made  to  integrate  the  automated 
instruction  in  the  most  appropriate  manner  with  the  other  aspects  of  the  program,  such  as  the 
practical  or  laboratory  problems  and  the  study  halls.  That  students  were  washed-back  from  most 
groups  indicates  that  there  is  room  for  improvement  in  all  the  techniques  used  in  either  selecting 
or  teaching  individuals. 

Further  research  on  auto-instructional  techniques  and  materials,  on  their  implications  for 
curriculum  planning,  and  details  of  school  administration  is  urgently  needed.  Additional  data 
should  be  collected  using  the  available  machines  in  combination  with  modified  materials  and 
administrative  procedures.  A  study  involving  one  such  modification — programmed  material  in 
the  form  of  scrambled  books — is  in  progress  and  will  be  reported  subsequently,  as  will 
studies  using  students  from  the  top  and  bottom  one-fifth  of  the  distribution  on  the  Keesler  Math 
Test.  Improved  versions  of  both  the  programmed  learning  material  and  the  presentation  device 
have  already  been  developed  by  the  contractor  supplying  the  materials  used  in  this  study.  These 
and  other  devices  now  on  the  commercial  market,  as  well  as  other  programming  formats,  should 
be  explored  for  their  potential  for  enhancing  the  efficiency  of  auto-instruction  beyond  that  observed 
here.  In  short,  this  study  should  be  interpreted  merely  as  a  demonstration  of  the  feasibility  of 
auto- instruction.  It  does  not  provide  a  conclusive  estimation  of  the  rela.ive  merits  of  auto- 
instructional  techniques  as  compared  with  conventional  techniques. 
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APPENDIX 
TABLE  VIII 

RESEARCH  STUDY  A  -  SUMMARY 


Average  Grade 

Elec 

Keesler 

Error 

Subject 

Blocks  I,  II  &  III 

AI 

EPSAT 

Math  Test 

Ratio 

Time  (Min.) 

AEB 

92 

85 

78 

60 

7.62 

3798.6 

AED 

88 

85 

84 

66 

3.85 

3195.5 

AEP 

84 

70 

59 

50 

2.67 

3659.9 

< 

AEH 

84 

85 

73 

58 

3.55 

4077.1 

AEE 

84 

75 

58 

56 

2.59 

3782.0 

w 

s 

AEK 

84 

75 

97 

62 

3.14 

3090.0 

AEC 

83 

70 

48 

68 

2.70 

4176.5 

as 

AEL 

81 

60 

92 

58 

3.04 

2525.8 

g 

AEN 

81 

75 

26 

54 

1.98 

3391.8 

a 

w 

AEF 

81 

75 

48 

58 

2.43 

3782.7 

AEO 

80 

80 

64 

52 

3.41 

4009.6 

AEA 

80 

70 

40 

70 

3.22 

3411.2** 

AEG 

79 

70 

64 

64 

2.76 

1574. 6** 

AEM 

70 

70 

31 

68 

2.43 

1182.3* 

MEAN 

82.21 

74.64 

61.57 

60.29 

ACA 

91 

95 

108 

54 

ACF 

90 

85 

79 

58 

ACD 

88 

75 

54 

70 

ACH 

86 

75 

60 

66 

ACK 

85 

85 

84 

52 

o 

ACL 

85 

80 

49 

62 

OS 

ACN 

85 

75 

63 

68 

H 

ACO 

85 

75 

41 

58 

o 

ACM 

84 

80 

58 

66 

u 

ACB 

83 

75 

54 

60 

ACE 

83 

75 

52 

56 

ACG 

82 

95 

52 

48 

ACP 

82 

85 

55 

58 

ACC 

70 

80 

31 

50 

MEAN 

84.21 

81.07 

60.00 

59.00 

ABK 

91 

90 

64 

ABC 

89 

80 

56 

ABB 

88 

85 

58 

ABD 

88 

85 

68 

ABO 

88 

85 

62 

fc 

ABG 

87 

80 

66 

o 

ABH 

86 

80 

52 

u 

ABN 

85 

75 

50 

o 

ABM 

84 

85 

56 

ABL 

83 

80 

68 

« 

ABE 

82 

75 

54 

ABP 

81 

70 

62 

ABA 

72 

85 

72 

ABF 

72 

80 

54 

MEAN 

84.00 

81.07 

60.14 

*  Block  I  only  on  AutoTutor  Elec  AI  -  Electronic  Aptitude  Index 

♦♦Blocks  I  and  n  only  on  AutoTutor  EPSAT  -  Engineering  and  Physical  Science  Aptitude  Test 
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TABLE  IX 


RESEARCH  STUDY  A  -  SUMMARY  OF  BLOCK  GRADES 


Subject 

Block  I 

Block  II 

Block  HI 

Mean 

Practical 

Theory 

Final 

Practical 

Theory 

Final 

Practical 

Theory 

Final 

AEA 

78 

78 

78 

75 

75 

75* 

AEB 

87 

88 

88 

89 

98 

96 

90 

92 

91 

91.67 

AEC 

90 

82 

84 

85 

80 

81 

81 

84 

83 

82.67 

AED 

81 

92 

89 

86 

87 

87 

83 

90 

88 

88.00 

AEE 

80 

83 

82 

78 

83 

82 

86 

85 

85 

83.00 

55 

AEF 

84 

80 

81 

73 

80 

79 

74 

77 

76* 

78.67 

Jg 

AEG 

70 

81 

79 

74 

75 

75* 

K 

AEH 

86 

85 

85 

80 

83 

82 

80 

87 

85 

84.00 

W 

AEK 

79 

82 

81 

81 

83 

83 

88 

86 

87 

83.67 

S 

AEL 

88 

79 

81 

90 

84 

85 

90 

73 

78 

81.33 

U 

AEM 

72 

75 

75* 

AEN 

85 

81 

82 

76 

78 

78 

83 

85 

84 

81.33 

AEO 

86 

85 

85 

87 

76 

78 

80 

77 

78 

80.33 

AEP 

79 

85 

84 

84 

87 

86 

80 

84 

83 

84.33 

MEAN 

81.79 

82.57 

82.43 

81. 

38 

82.23 

82.08 

83. 

18 

83.64 

83.45 

ACA 

90 

95 

94 

89 

89 

89 

90 

92 

91 

91.33 

ACB 

86 

83 

83 

82 

75 

76* 

ACC 

76 

70 

71* 

ACD 

79 

90 

88 

87 

87 

87 

92 

87 

89 

88.00 

ACE 

70 

83 

80 

85 

85 

85 

81 

86 

85 

83.33 

o 

ACF 

83 

92 

90 

86 

88 

88 

90 

92 

91 

89.67 

g 

ACG 

83 

82 

82 

86 

73 

77* 

r* 

ACH 

91 

86 

87 

80 

88 

86 

80 

85 

84 

85.67 

o 

u 

ACK 

88 

85 

86 

89 

85 

86 

83 

83 

83 

85.00 

ACL 

85 

90 

89 

86 

82 

83 

82 

84 

83 

85.00 

ACM 

86 

82 

83 

87 

83 

84 

85 

85 

85 

84.00 

ACN 

79 

86 

85 

89 

86 

87 

83 

84 

84 

85.33 

ACO 

83 

88 

87 

81 

80 

80 

81 

90 

87 

84.67 

ACP 

84 

82 

82 

84 

70 

73* 

MEAN 

83.07 

85.28 

84.78 

85.46 

82.38 

83.15 

84. 

70 

86.80 

86.20 

ABA 

84 

84 

84 

77 

70 

71* 

ABB 

85 

88 

87 

87 

92 

91 

88 

87 

87 

88.33 

ABC 

80 

95 

92 

86 

89 

88 

82 

89 

87 

89.00 

o 

ABD 

79 

92 

89 

75 

91 

88 

82 

91 

88 

88.33 

« 

ABE 

85 

84 

84 

86 

81 

82 

70 

78 

76* 

80.67 

H 

55 

ABF 

88 

88 

88 

80 

79 

79 

80 

71 

74* 

80.33 

o 

ABG 

87 

88 

88 

90 

87 

88 

86 

83 

84 

86.67 

O 

ABH 

88 

92 

91 

80 

86 

85 

86 

80 

82 

86.00 

Q 

5 

j 

ABK 

90 

95 

94 

87 

87 

87 

88 

94 

92 

91.00 

ABL 

85 

88 

87 

83 

76 

77* 

CQ 

ABM 

88 

85 

86 

86 

83 

84 

76 

74 

75* 

81.67 

ABN 

88 

87 

87 

81 

86 

85 

84 

83 

83 

85.00 

ABO 

86 

95 

93 

80 

87 

87 

88 

84 

85 

88.33 

ABP 

84 

85 

83 

86 

77 

79 

80 

81 

81 

81.00 

MEAN 

85.50 

89.00 

88.07 

83.14 

83.64 

83.64 

82.50 

82.92 

82.83 

♦Wash-back 
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TABLE  X 

RESEARCH  STUDY  A  -  TABLES  OF  CORRELATIONS 
Variables: 

1-  Keesler  Math  Test 

2.  Average  Blocks  L  IL  and  III  Final  Grades 

3.  Electronic  Aptitude  Index 

4.  Engineering  and  Physical  Science  Aptitude  Test 

5.  Time 

6.  Error  Ratio 


1 

2 

3 

4 

5 

6 

1  2 

3 

1 

1  -  .05 

.39 

2 

-  .25 

2 

.25 

3 

-  .12 

.49* 

3 

4 

-  .10 

.62** 

.19 

Blind  Control  Group  (N=  14) 

5 

-  .06 

.05 

.47 

-  .51 

6 

.06 

.68** 

.54* 

.45 

.10 

Experimental 

Group 

(N  » 

11  to  14) 

1 

2 


2  3  4 

39  -.65*  -.07 

.15  .71** 


3 


.55* 


4 


Control  Group  (N=14) 


•Sign  5% 
•♦Sign  1% 
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EXPERIMENTAL 
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TABLE  XI 


RESEARCH  STUDY  B  -  SUMMARY 


Subject 

Average  Grade 
Blocks  l  n  &  m 

Elec 

AI 

EPSAT 

Keesler 

Math  Test 

Error 

Ratio 

Time  (Min. 

BEE 

90 

90 

97 

64 

5.20 

2703 

BEG 

89 

85 

70 

90 

3.89 

3600 

BEK 

87 

90 

78 

68 

4.18 

2865 

BEM 

84 

75 

32 

60 

3.02 

4061 

BEL 

84 

85 

50 

68 

2.82 

3948 

BEN 

82 

80 

57 

78 

2.96 

3191 

BEF 

82 

75 

56 

82 

3.46 

3074** 

BEA 

82 

70 

48 

54 

3.26 

3909** 

BEO 

82 

70 

72 

76 

3.33 

3432 

BEP 

81 

85 

71 

60 

2.54 

3991 

BED 

80 

95 

82 

78 

2.59 

4469 

BEH 

80 

75 

38 

62 

1.93 

2275** 

BEB 

79 

75 

52 

58 

2.51 

2379** 

BEC 

78 

70 

29 

66 

2.35 

1386* 

MEAN  82.86  80.00  59. 43  68.86 


BCO 

94 

75 

53 

72 

BCK 

89 

95 

48 

74 

BCE 

88 

80 

51 

78 

BCA 

87 

80 

48 

60 

BCP 

87 

70 

-- 

56 

►J 

o 

BCB 

87 

70 

-- 

64 

P5 

BCF 

86 

70 

48 

52 

Z 

BCD 

86 

75 

64 

58 

o 

BCG 

84 

80 

70 

80 

BCL 

83 

70 

55 

86 

BCH 

83 

70 

72 

68 

BCN 

83 

70 

61 

62 

BCC 

81 

75 

-- 

60 

BCM 

81 

70 

41 

68 

MEAN 

85.64 

75.00 

55.54 

67.00 

BBD 

94 

80 

90 

BBH 

90 

75 

61 

54 

BBN 

90 

95 

75 

82 

►J 

BBP 

89 

80 

72 

66 

K 

BBG 

89 

75 

68 

68 

H 

BBM 

89 

80 

-- 

78 

O 

BBO 

88 

75 

-- 

78 

U 

BBA 

87 

85 

48 

76 

o 

BBC 

86 

80 

76 

66 

BBB 

86 

85 

39 

64 

la 

BBL 

83 

80 

37 

62 

BBF 

81 

80 

57 

58 

BBK 

81 

80 

41 

62 

BBE 

80 

80 

62 

60 

MEAN 

86.64 

80.71 

57.82 

68.86 

♦Block  I  only  on  AutoTutor  ElecAI  -  Electronic  Aptitude  Index 

♦♦Blocks  I  and  II  only  on  AutoTutor  EPSAT  -  Engineering  and  Physical  Science  Aptitude  Test 
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TABLE  XII 

RESEARCH  STUDY  B  -  SUMMARY  OF  BLOCK  GRADES 


Subject 

Block  I 

Block  II 

Block  III 

Mean 

Practical 

Theory 

Final 

Practical 

Theory 

Final 

Practical 

Theory 

Final 

BEA 

76 

79 

78 

82 

78 

79** 

BEB 

80 

79 

79 

79 

71 

73* 

BEC 

70 

70 

70* 

< 

BED 

82 

82 

82 

81 

80 

80 

80 

74 

76* 

79.33 

H 

5PI 

BEE 

92 

90 

90 

82 

89 

88 

92 

94 

93 

90.33 

u 

BEF 

70 

82 

80 

79 

76 

77* 

2 

BEG 

92 

90 

90 

89 

88 

88 

84 

90 

88 

88.67 

2 

u 

Di 

BEH 

85 

81 

82 

82 

73 

75* 

BEK 

89 

85 

86 

81 

88 

87 

88 

89 

89 

87.33 

X 

ca 

BEL 

87 

83 

84 

86 

81 

82 

83 

86 

85 

83.67 

BEM 

79 

88 

86 

76 

83 

82 

86 

85 

85 

84.33 

BEN 

82 

82 

82 

79 

82 

81 

77 

87 

84 

82.33 

BEO 

80 

88 

86 

78 

81 

80 

80 

79 

79 

81.67 

BEP 

89 

81 

83 

85 

79 

80 

80 

78 

79 

80.67 

MEAN 

82.36 

82.86 

82.71 

81.46 

80. 

69 

80.92 

83.33 

84.67 

84.22 

BCA 

94 

88 

89 

89 

85 

86 

92 

85 

87 

87.33 

BCB 

92 

86 

87 

84 

89 

88 

88 

83 

85 

86.67 

BCC 

76 

82 

81 

72 

83 

81 

86 

80 

82 

81.33 

BCD 

82 

88 

87 

82 

89 

88 

77 

84 

82 

85.67 

BCE 

92 

95 

94 

84 

86 

86 

81 

86 

85 

88.33 

O 

BCF 

82 

95 

92 

80 

85 

84 

64 

83 

83 

86.33 

05 

BCG 

79 

85 

84 

79 

83 

82 

80 

89 

86 

84.00 

z 

BCH 

84 

78 

79 

83 

84 

84 

92 

82 

85 

82.67 

o 

V 

BCK 

87 

88 

88 

89 

88 

88 

84 

96 

92 

89.33 

BCL 

83 

86 

85 

81 

80 

80 

84 

85 

85 

83.33 

BCM 

82 

81 

81 

82 

83 

83 

82 

79 

80 

81.33 

BCN 

81 

86 

85 

85 

81 

82 

84 

80 

81 

82.67 

BCO 

94 

95 

95 

90 

94 

93 

92 

96 

95 

94.33 

BCP 

79 

86 

85 

83 

88 

87 

88 

90 

89 

87.00 

MEAN 

84.79 

87. 

07 

86.57 

83 , 07 

85. 

57 

85.14 

85.29 

85.57 

85.50 

BBA 

80 

88 

86 

82 

86 

85 

86 

91 

90 

87.00 

BBB 

81 

91 

89 

85 

85 

85 

86 

83 

84 

86.00 

BBC 

85 

86 

86 

83 

89 

88 

80 

87 

85 

86.33 

o 

BBD 

92 

93 

93 

90 

95 

94 

92 

96 

95 

94.00 

OS 

BBE 

77 

82 

81 

79 

79 

79 

84 

79 

81 

80.33 

g 

BBF 

70 

85 

82 

79 

81 

81 

84 

78 

80 

81.00 

o 

BBG 

80 

91 

89 

86 

87 

87 

84 

93 

90 

88.67 

u 

BBH 

83 

93 

91 

82 

94 

92 

82 

89 

87 

90.00 

9 

BBK 

78 

83 

82 

80 

79 

79 

81 

75 

77* 

79.33 

2 

BBL 

85 

81 

82 

82 

84 

84 

81 

84 

83 

83.00 

n 

BBM 

81 

89 

87 

89 

91 

91 

80 

91 

88 

88.67 

BBN 

77 

91 

88 

90 

92 

92 

85 

91 

89 

89.67 

BBO 

79 

91 

89 

83 

86 

85 

90 

91 

91 

88.33 

BBP 

87 

91 

90 

79 

87 

85 

88 

93 

92 

89.00 

MEAN 

81.07 

88. 

21 

86.79 

83.50 

86.79 

86.21 

84.50 

87.21 

86.57 

♦Wash-back 
♦♦Emergency  Leave 
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TABLE  XIII 

RESEARCH  STUDY  B  -  TABLES  OF  CORRELATIONS 


1IRL-TDR-62-78 


Variables: 

1. 

Keesler  Math  Test 

2. 

Average  Blocks  L  II,  and  III  Final  Grades 

3. 

Electronic  Aptitude  Index 

4. 

Engineering  and  Physical  Science  Aptitude  Test 

5. 

Time 

6. 

Error  Ratio 

1 

2 

3 

4  5  6 

1  2 

3 

4 

1  . 70**  . 

34 

.28 

.28 

2 

00 

.58* 

.29 

.53* 

3 

.07 

.33 

.57* 

.73** 

4 

.01 

-  .52* 

.01 

-  .44 

Blind  Control  Group  (N  =  11 

to  14) 

.23 

.89** 

.38 

.67**  -  .82** 

Experimental  Group  (N  =  14) 


1  2 

3 

4 

1 

.10 

.32 

.12 

2 

.41 

o 

CM 

i 

3 

-.17 

4 


Control  Group  (N  -  11  to  14) 


•Sign  5% 
♦•Sign  1% 


22 


-e- 


a  3 

.a  M  S 

>  c  'S 
■81  u 


CO 
w 
CO 

c  ” 

I  "S"5  .>S  fc 

*  '5*  13  £*  g,  Si  < 

o  £  6  o  ®  3  «  t5 

-  , z  1 2  “  °  e 

-  _  i»55faHe3M. 

Hf£<-§<<£o!).3hB:oSp5! 

a  a  > 


1> 


